4D Temporally Coherent Multi-Person Semantic Reconstruction and Segmentation

Abstract We introduce the first approach to solve challenging problem of automatic 4D visual scene understanding for complex dynamic scenes with multiple interacting people from multi-view video. Our simultaneously estimates a detailed model that includes per-pixel semantically and temporally coherent reconstruction, together instance-level segmentation exploiting photo-consistency, semantic motion information. further leverage recent advances in 3D pose estimation constrain joint instance reconstruction. This enables per person scenes. Extensive evaluation framework against state-of-the-art methods on indoor outdoor sequences demonstrates significant ( $$\approx 40\%$$ ≈ 40 % ) improvement segmentation, reconstruction flow accuracy. In addition several scenes, proposed is applied sports wild captured manually operated wide-baseline broadcast cameras.